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ABSTRACT 

The long-term effects of Professional Development (PD) on educators ’ ability’ to affect student 
performance remain virtually unstudied. This quasi-experimental, longitudinal study compared 
student achievement scores from 25 states and 78 school districts over multiple years for schools 
whose teachers used an online, on-demand professional development offering. The objective was 
to evaluate to what degree student scores might improve, decline, or plateau with participation in 
professional development over multiple years. Results showed significant gains in student 
performance across the number of years of participation, with significant continuation in growth 
across years. Findings validate a significant predictive relationship between the number of years 
educators participated in PD and improvements quantified in student performance. Inferences of 
findings to other PD approaches or offerings, or other student or educator populations or settings 
are discussed. 

Keywords: Professional Development; Multivariate Statistical Methods; On-Demand Professional 
Learning; Long-Term 


BACKGROUND & REVIEW OF THE LITERATURE 

Professional development (PD) for teachers has long been a key and logically positioned undertaking for 
improving teacher satisfaction, as well as hope for improved classroom instruction and student achievement (Ball & 
Cohen, 1999; Cohen & Hill, 2000; Corcoran, Shields, & Zucker, 1998; Darling-Hammond & McLaughlin, 1995; 
Elmore, 1997; Little, 1993; National Commission on Teaching and America’s Future, 1996; Wayne et ah, 2008; 
Yoon et ah, 2007). However, calls for high-quality PD remain persistent, yet there is a continued shortage of proven 
programs, including PD offerings that are data-verified or characterized as coherent, active learning, of sufficient 
duration, content knowledge-focused, or reflective of reform (Birman et ah, 2007; Garet et al, 2001; Loucks-Horsley 
et ah, 1998; Shaha & Ellsworth, 2014; U.S. Department of Education, 2001). 

The single greatest criticism of PD is the overwhelming prevalence of single-shot, one-day workshops that 
often make teacher professional development “intellectually superficial, disconnected from deep issues of 
curriculum and learning, fragmented, and noncumulative” (Ball & Cohen, 1999, pp. 3—4). Many argue that coherent 
infrastructure for PD and its evaluation is either absent or too limited and thus PD represents a “patchwork of 
opportunities —formal and informal, mandatory and voluntary, serendipitous and planned’’ (Wilson & Berne, 1999, 
p.174). Such a critique is untenable for long-term or broad and generalizable PD success. 

There is an unfortunate and unjustifiable lack of scientifically-based research that demonstrates quantified 
PD-driven improvement in student achievement. Substantive information and rigorous data on how PD programs 
affect student achievement is an urgent need. Yet, data-proven research in PD efficacy remains rare, reflecting 
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statistically rigorous studies demonstrating PD impact on teacher efficacy toward actual student achievement 
(Fishman et al, 2003; Loucks-Horsley et al., 2003; Shaha et al., 2004; Shaha & Ellsworth, 2014; Wilson & Berne, 
1999). Due to the lack of research, PD purchases continue to rely heavily or exclusively on unsubstantiated 
marketing materials or testimonials. 

Despite all observations regarding PD and criticisms, several recent studies have verified significant growth 
in student performance associated with teacher participation in PD, particularly in online, on-demand offerings 
(Avalos, 2011; Buczynski & Hansen, 2010; Desimone et al., 2002; Ingvarson et al., 2005; Shaha et al., 2004; Shaha 
& Ellsworth, 2013; Wasik & Hindman, 2011). Further, results from a handful of specific studies establish that the 
degree of teacher engagement in PD significantly affects the amount of improvement experienced in student 
performance, contrasting active participation versus more passive engagement based principally on video 
participation alone (Desimone et al., 2002; Darling-Hammond, 2004; Garet et al., 2001; King, 2002; Santagata, 
2009; Shaha & Ellsworth, 2014). 

Therefore, data-backed scientific research has begun but needs to grow. What our industry needs are 
rigorous, substantive studies that establish the credibility and impact of PD offerings and participation. Furthermore, 
such proof should mature beyond a single year of impact and verify that gains are held over time, or even improved 
longitudinally, with ongoing PD participation or improved use and implementation. 

The purpose of this research was to measure whether continued participation in PD over multiple years 
would have long-term, continued benefits in student achievement or whether impacts would plateau over time or 
even diminish. The goal was to quantify whether student gains were significant and sustained over time with 
continued teacher PD participation. 

METHODS 

The study design reflected a quasi-experimental, longitudinal design (Cook & Campbell, 1979). School 
participation was defined as use of a single commercially available online, on-demand PD product widely used in 
the United States (PD 360 ®, Common Core 360®, and/or Observation 360 ®). This enabled teachers to engage in a 
breadth of activities ranging from instructional videos on teaching techniques to higher levels of participation, such 
as professional learning communities and posting and downloading PD-related materials. 

Student performance data were gathered from publically available web sources. To overcome variations in 
assessment instruments across states and districts, comparative performance was analyzed as changes in the 
percentage of students classified as either “advanced” or “proficient” regardless of the test (no within-state changes 
in testing were noted within the sample for the data). Districts in the study met minimal inclusion requirements of an 
average of 90 minutes of voluntary PD participation annually, and only districts with more than a single year of PD 
participation were included. Data were aggregated into two-year increments for analysis to maximize sample sizes 
for statistical comparisons. All comparisons contrasted net gain in student performance from baseline or the last year 
pre-implementation. 

All analyses were conducted using SPSS version 17.0 or higher (PASW Statistics; SPSS, 2009, with SAS 
used for confirmatory purposes when results were close to p<0.05). Minimum level of statistical significance was 
determined a priori at p<0.05. To ensure conservative interpretations, the n for each statistical test was held to the 
number of districts and not to schools, educators, or students. 

RESULTS & FINDINGS 

Final data included 78 districts from 25 states. Longevity among districts included ranged from two to 
seven years. The demographic distribution of the sample by Urban and Title I schools is shown in Table 1. 
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Table 1: Demographic Mix By Years Of PD Participation 


Urban 

Title 1 


N 

Pet 

p-Value 

Pet 

p-Value 

2-3 years 

22 

18.2 % 


61.0% 


4-5 years 

45 

33.3% 

0.037 4-5 > 2-3, 6-7 

72.0% 


6-7 years 

11 

18.2% 


84.0% 

0.038 6-7>2-3 


The comparative net change in mean student performance versus baseline showed consistent and 
significant growth across longitudinal groupings for both reading and math (see Table 2 and Figure 1). For reading 
in schools that had participated for only two to three years, there was an increase of 5.58 percent of students 
proficient or advanced. For the group participating four to five years, the increase in performance was 7.34 
(p<0.042), and for six to seven years, performance increased by 12.77 percentage points (p<0.037). 


Table 2: Net Change In Student Performance By Years Of PD Participation 


Reading Performance 

Math Performance 


N 

Net Mean Change 

p-Value 

Net Mean Change 

p-Value 

2-3 years 

22 

5.58 


2.90 


4-5 years 

45 

7.34 

p=0.042 

6.92 

p=0.031 

6-7 years 

11 

12.77 

p=0.037 

12.14 

p=0.009 
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Longitudinal Effect on Change in Student Performance 
Versus Pre-PD Baselines 
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Figure 1: Net Change In Student Performance By Years Of PD Participation 


For mathematics, the net mean change in student performance levels for schools that participated for two to 
three years was an increase of 2.90 in percent of students classified as proficient or advanced. For the group 
participating four to five years, the increase in performance was 6.92 (p<0.031), and for six to seven years, 
performance increased by 12.14 percentage points (p<0.009). 


CONCLUSIONS 


Districts in 25 states whose teachers actively participated in the online, on-demand PD offering experienced 
significant improvement in student achievement. Equally important, the longer the participation was maintained, the 
greater the growth that was experienced and sustained. Clearly, the data show that the longer teachers participated in 
the PD test, the greater the gains they experienced for their students. The degree of improvement leads to an 
interpretation that longer engagement is causal, rather than an alternative inference that continued improvement was 
causal to continued engagement. 

Data from studies of PD consistently encourage that every educational institution spending on PD should 
ensure that such represents a genuine investment. Investments infer data wherein either the statistically proven 
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successes of others or from within are incorporated into decision making to ensure data-driven assurances of 
promises beyond marketing tracts alone. In the absence of proof, money used represents only an expenditure or an 
outlay without substantive evidence of impact. The measure of impact cannot be based upon the happiness levels of 
participating teachers alone, as might be the case for in-person PD seminars or workshops. Rather, the measure of 
success should include or be expressed in impact upon students for better meeting the goals toward which teachers 
and schools are aimed. Spending on unmeasured PD remains a risk—PD is NOT an investment until it is evaluated 
and quantified and proven to impact teacher and school efficacy. 

Whether these results are exclusive to this specific PD offering remains an open question. The answer will 
remain unknown until similar rigorous, statistically-backed studies are executed for other PD offerings and 
implementations. For now, this remains the only multi-state study in existence according to an exhaustive review of 
the literature, providing a basis in methodology and findings upon which comparative studies should be executed. 

In conclusion, the efficacy and impact of teachers participating in PD can be, and should be, shown in 
readily interpretable statistical results. Educators continuously seek greater impact on students and PD can now be 
touted as an affirmative action in that crucial direction. Additionally, it is comforting that the lower-cost, more 
readily accessible capabilities of high engagement use of an online, on-demand PD offering has near-term and 
lasting impacts on the students relying on teachers. 
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